sigstore: Cache verify_image results across policy evaluations#3136
Conversation
3950ea8 to
b0710ba
Compare
b0710ba to
d029230
Compare
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Review Summary by QodoCache verify_image results across policy evaluations
WalkthroughsDescription• Add caching layer for verify_image results using sync.Map and singleflight.Group • Cache key combines image reference and SHA256 hash of verification options • Prevents redundant signature verification across multiple policy evaluations • Add comprehensive test coverage for cache hit/miss scenarios Diagramflowchart LR
A["sigstoreVerifyImage called"] --> B{"Cache hit?"}
B -->|Yes| C["Return cached result"]
B -->|No| D["Use singleflight dedup"]
D --> E["doVerifyImage execution"]
E --> F["Store result in cache"]
F --> G["Return result"]
C --> H["Result"]
G --> H
File Changes1. internal/rego/sigstore/sigstore.go
|
Code Review by Qodo
1. Caches failed verifications
|
d029230 to
c504a19
Compare
|
Here's the performance stats I ran. We should try and look for improvements. If we can't, I think we'll have to send out communication on having to up memory requirements. Current PRPeak RSS (MB)
Wall Time (s)
main branchPeak RSS (MB)
Wall Time (s)
|
|
@joejstuart , thanks for that analysis. Is this increase just if we are adding the signature verification? So users would not need to increase memory requirements unless they used that functionality? If we were trying to keep ~200 components to finish evaluation in under 5 minutes, then we also fail that goal with the associated policy changes. |
I tested it with the corresponding policy change. Should I test without it? |
|
I think we needed the test with the policy change to understand the impact of that change. But it would be good to test it without as well to ensure that we don't have regression when that policy isn't being used. |
|
Without the policy change Peak RSS (MB)
Wall Time (s)
|
|
Thanks, @joejstuart . These are just single runs or averages over multiple runs? |
|
@joejstuart @simonbaird , would you be okay with this change along with conforma/policy#1680 ? |
|
Now I'm not seeing much of a performance difference between this and |
|
/ok-to-test |
c8a13b9 to
c97787f
Compare
📝 WalkthroughWalkthroughThe PR adds verification result caching to ChangesSigstore verification caching
🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Rebased to see if it gets closer to green. |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
internal/rego/sigstore/sigstore.go (1)
480-483: ⚡ Quick win
ClearCachesreassigning thesync.Mapvalue is racy.
verifyImageCacheis a package-level value (not a pointer); reassigning it while other goroutines may still be callingLoad/Storeon it is a data race. It happens to work in serial test runs, butgo testcan run tests from other packages or the same package in parallel, and the singleflight callback above also touches it from inner goroutines. Prefer clearing in place.-func ClearCaches() { - verifyImageCache = sync.Map{} -} +func ClearCaches() { + verifyImageCache.Range(func(k, _ any) bool { + verifyImageCache.Delete(k) + return true + }) +}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/rego/sigstore/sigstore.go` around lines 480 - 483, ClearCaches currently reassigns the package-level sync.Map verifyImageCache which is racy; instead clear the map in place to avoid races: in ClearCaches iterate the existing verifyImageCache (use verifyImageCache.Range) and delete each key (verifyImageCache.Delete) so any concurrent Load/Store operations remain safe; update the ClearCaches function to perform in-place deletion rather than reassigning verifyImageCache.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/rego/sigstore/sigstore_test.go`:
- Around line 224-241: The newFakeContext helper misses a mock for
fake.FakeClient.HasBundles which doVerifyImage calls before
VerifyImageSignatures; update newFakeContext to add an expectation on c (the
*fake.FakeClient) for HasBundles (use mock.Anything for args) and return the
same boolean/error used in other tests (e.g., false, nil) so the HasBundles call
is explicitly verified before VerifyImageSignatures is exercised.
In `@internal/rego/sigstore/sigstore.go`:
- Around line 140-149: The current wrapper unconditionally stores the term
returned by doVerifyImage in verifyImageCache via verifyImageCache.Store,
causing transient verification failures to become permanently cached; change the
flow so only cacheable (deterministic/success) results are stored: have
doVerifyImage indicate cacheability (e.g., return a second bool or make the
ast.Term include a "success"/"cacheable" flag) and update the
verifyImageFlight.Do callback/result handling to inspect that flag and call
verifyImageCache.Store only when cacheable/successful; keep verifyImageFlight.Do
and verifyImageCache usage but skip storing non-cacheable failure terms so
transient errors are not persisted.
---
Nitpick comments:
In `@internal/rego/sigstore/sigstore.go`:
- Around line 480-483: ClearCaches currently reassigns the package-level
sync.Map verifyImageCache which is racy; instead clear the map in place to avoid
races: in ClearCaches iterate the existing verifyImageCache (use
verifyImageCache.Range) and delete each key (verifyImageCache.Delete) so any
concurrent Load/Store operations remain safe; update the ClearCaches function to
perform in-place deletion rather than reassigning verifyImageCache.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Enterprise
Run ID: 2a574443-5227-40c2-9e2c-e9fffd56ad98
📒 Files selected for processing (2)
internal/rego/sigstore/sigstore.gointernal/rego/sigstore/sigstore_test.go
|
/ok-to-test |
|
@st3penta , did you move this to draft because there is something conflicting or just because there hasn't been much review volume? Are there any concerns with merging this? |
|
hi @arewm, i moved it to draft because it seemed a bit stale, but talking with the team we agreed we should instead get it green and reviewed, and merge it |
|
I rebased on all the commits from #3357 to see if it goes green. Would probably want to rebase again on updated main once 3357 is merged. Update: Done now. |
|
🤖 Finished Review · ✅ Success · Started 7:59 PM UTC · Completed 8:12 PM UTC |
|
/ok-to-test |
975e565 to
0109754
Compare
|
🤖 Finished Review · ❌ Failure · Started 1:29 PM UTC · Completed 1:34 PM UTC |
Superseded by updated review
|
/ok-to-test |
0109754 to
6692db6
Compare
|
Addressing fullsend's comments:
|
|
/ok-to-test |
|
Should be good to merge. If it goes green, I'll seek an ack and aim to get it merged. |
Add a sync.Map + singleflight.Group cache for ec.sigstore.verify_image results keyed by ref + opts hash. OPA's Memoize only deduplicates within a single Eval() call, but components are validated in separate Eval() calls. Since bundles are pinned by digest, verification results are stable and can be safely cached for the process lifetime. This prevents redundant signature verification when many components share the same task bundles (e.g., 100 components using git-clone). Ref: EC-1545 Assisted-by: Claude Code (Sonnet 4.6) Signed-off-by: arewm <arewm@users.noreply.github.com>
6692db6 to
e145df4
Compare
|
/ok-to-test |
|
I just noticed there were some other fullsend suggestions, largely related to explict cache expiry. I think they're reasonable, but not high value. Also I think we've spent enough energy on this now, so I closed the threads. |
|
Let's merge. |
|
🤖 Finished Retro · ✅ Success · Started 8:43 PM UTC · Completed 8:53 PM UTC |
|
PR #3136 (conforma/cli) added caching for sigstore verify_image results. The fullsend review agent provided the highest-impact finding: missing ClearCaches() calls in bundle-related tests causing them to silently pass via cached results. This was fixed. The main friction: all three bots (qodo, coderabbit, fullsend) repeatedly flagged the same design tradeoff (caching transient failures), which the author had explicitly acknowledged. Fullsend re-raised this as [high] logic-error across 2 runs. The PR was merged with fullsend reviews DISMISSED and no review on the final code after fixes were applied. Proposals filed
|
Summary
sync.Map+singleflight.Groupcache forec.sigstore.verify_imageresults keyed byref|hash(opts)manifestCache/descriptorCacheininternal/rego/oci/oci.goTest plan
go build ./internal/rego/sigstore/compiles cleanlyRef: EC-1545
🤖 Generated with Claude Code